The AI appears to like paging and reading more detailed websites, however, I think there should be an option to obtain a get_text_summary of the whole webpage via prompt.

@Agusx1211 Could you please resubmit this as an optional (using an argument) rather than replacing the existing code?

There have been some good improvements to browsing since this was submitted, but we're still looking for a major solution.

I can't work on this until late night. What do you mean as an optional? like adding a flag that changes the behavior?

I can't work on this until late night. What do you mean as an optional? like adding a flag that changes the behavior?

Yes, looks like he meant this. Cause it's major improvement which needs separate flag

See also PR #96 which uses Playwright instead of requests.

But I also second the request that this is triggered by an argument.

I think this should be an alternate kind of command that the AI can run if the default (static) beautifulsoup parser fails.

An important thing is that Selenium makes it harder to setup. And it somehow assumes that Chrome is the only Selenium web driver. So, for it to merge I guess (I'm not the maintainer) it should be an entirely optional feature with a separate document describing the installation, keeping the current "for dummies" installation procedure as is.

Selenium is now in the master branch